A hierarchical indexing strategy for optimizing Apache Spark with HDFS to efficiently query big geospatial raster data     

Fei Hu  Yongyao Jiang  Yun Li  Weiwei Song  Daniel Q. Duffy 《International Journal of Digital Earth》2020,13(3):410-428

ABSTRACT

Earth observations and model simulations are generating big multidimensional array-based raster data. However, it is difficult to efficiently query these big raster data due to the inconsistency among the geospatial raster data model, distributed physical data storage model, and the data pipeline in distributed computing frameworks. To efficiently process big geospatial data, this paper proposes a three-layer hierarchical indexing strategy to optimize Apache Spark with Hadoop Distributed File System (HDFS) from the following aspects: (1) improve I/O efficiency by adopting the chunking data structure; (2) keep the workload balance and high data locality by building the global index (k-d tree); (3) enable Spark and HDFS to natively support geospatial raster data formats (e.g., HDF4, NetCDF4, GeoTiff) by building the local index (hash table); (4) index the in-memory data to further improve geospatial data queries; (5) develop a data repartition strategy to tune the query parallelism while keeping high data locality. The above strategies are implemented by developing the customized RDDs, and evaluated by comparing the performance with that of Spark SQL and SciSpark. The proposed indexing strategy can be applied to other distributed frameworks or cloud-based computing systems to natively support big geospatial data query with high efficiency.  相似文献   

基于GNSS方位辅助惯性导航系统的水下地形精密测量技术     

冯义楷  丁继胜  杨龙  陈义兰  梁冠辉 《海洋技术学报》2019,38(5)

多波束测深技术是目前水下地形测量的主要技术手段,测量平台的瞬时姿态及方位是影响多波束测深系统最终成果准确度的重要因素。GNSS方位辅助惯性导航系统,作为目前应用较为广泛的方位、姿态、及位置综合测量系统,不仅能够提供高精度位置信息,同时也能提供测量平台的瞬时姿态及方位数据,而且因为具有GNSS方位辅助测量,使得最终方位测量结果比传统方位测量精度大大提高,这对于多波束最终测量成果精度提高具有重要意义。文中从GNSS方位辅助惯性导航系统原理及技术优势出发,结合Trimble RTX后处理技术,从姿态测量、方位测量及辅助高程测量方面分析了在多波束水下地形测量中的应用,并以实际测量成果来展现其在水下地形精密测量技术方面的优势,结果显示,定位精度可以达到优于2 cm级别,方位精度可以优于0.01°(依赖于双GNSS天线之间的基线长度),该技术对水下地形测量准确度提升作用显著。  相似文献   

无人机机载LiDAR在沿海滩涂大比例尺地形测绘中的应用     

郭春海  张英明  丁忠明 《测绘通报》2019,(9):155-158

以南京市以下长江12.5 m深水航道建设工程中的白茆沙整治建筑物工程的年度监测为例，介绍了利用无人机航摄和机载LiDAR技术进行大比例尺高精度的滩涂地理信息数据采集的关键技术。研究案例表明，采用无人机搭载数码相机和LiDAR联合进行数据采集，在小区域、通行困难地区的高分辨率影像和地面高程数据快速获取方面具有明显优势，为沿海滩涂开发中的工程用图测绘工作提供了一种新的有效快捷的技术手段。  相似文献   

基于2D地图的城市户外ARGIS视觉辅助地理配准技术     

邓晨  游雄  张威巍  智梅霞 《测绘学报》2019,48(10):1305-1319

针对当前便携式位姿传感器获取的户外6DOF绝对位姿精度通常不足，致使基于位姿传感器的户外AR地理配准精度不高，提出了一种以2D地图作为参考数据进行户外视觉辅助定位进而提高ARGIS系统地理配准精度的方法和技术框架。在位姿传感器获取初始位姿的基础上，详细阐述了基于2D地图进行图像位姿优化和辅助校正的基本原理；并通过试验验证了该方法可以对位姿传感器获取的初始位姿进行优化，进而提高户外AR地理配准精度的有效性。  相似文献   

机载WIDAS数据的Landsat卫星反照率初步验证   总被引：1，自引：1，他引：0  

何丹丹  焦子锑  董亚冬  张小宁  丁安心  尹思阳  崔磊  常雅轩 《遥感学报》2019,23(1):53-61

随着精细化监测的需求,中高空间分辨率的地表反照率产品逐渐成为气候模型的主要输入。目前,中高空间分辨率反照率产品的验证主要基于地表站点的通量塔观测数据,区域机载飞行数据的验证依然相对较少。因此,本文基于区域机载数据验证Landsat反照率产品。针对内蒙古自治区根河森林试验区所获取的机载红外广角双模式成像仪(WIDAS)多角度反射率数据,应用BRDF原型反演算法估算其反照率,分析了应用机载数据验证中高空间分辨率反照率产品的潜力。2016年内蒙古根河森林试验区机载WIDAS飞行多角度观测的可用多角度范围为25°,以前的研究表明BRDF原型反照率反演算法表现出对小观测角度的反照率反演结果的鲁棒性。因此,机载WIDAS反照率在一定程度可用于星载反照率的验证。首先,基于核驱动模型和各向异性平整指数(AFX)提取了试验区4种MODIS二向性反射分布函数(BRDF)原型;然后,将其作为先验知识应用到根河森林WIDAS机载数据的反照率反演中;最后,用WIDAS反照率和单个地面通量塔观测的反照率对Landsat卫星数据的反照率进行初步验证。验证结果表明Landsat反照率与WIDAS反照率结果较为一致,但略有低估,总体均方根误差(RMSE)约为0.02,偏差为0.0057。在多角度观测范围较小时,BRDF原型的反照率反演算法可为星载地表反照率的验证提供了一种有效的验证手段。  相似文献   

南海大洋钻探及海洋地质与地球物理前沿研究新突破   总被引：2，自引：2，他引：0  

林间  李家彪  徐义刚  孙珍  夏少红  黄小龙  解习农  李春峰  丁巍伟  周志远  张帆  罗怡鸣 《海洋学报》2019,41(10):125-140

南海是西太平洋地区规模最大且具有代表性的边缘海盆地之一。经过近几十年的研究积累，尤其是通过实施5个国际大洋钻探航次(1999–2018年)与国家自然科学基金委“南海深海过程演变”重大研究计划(2011–2019年)，我国科学家获得了大量宝贵的第一手资料，取得了一系列创新进展与重大突破，标志着南海海洋地质与地球物理研究正走向国际前沿。重要研究成果包括:(1)新提出南海是“板缘张裂”盆地，与经典的大西洋型陆缘模式不同；(2)大洋钻探首次获取了基底玄武岩样品，结合中国在南海首次深拖地磁测量实验，精确测定了南海海盆玄武岩年龄，揭示南海海盆从东向西分段扩张；(3)大洋钻探结果发现南海陆缘岩石圈减薄之初岩浆迅速出现，未发现缓慢破裂造成的蛇纹岩出露；(4)发现南海扩张结束后仍存在大量岩浆活动，可能受控于多种构造与地幔因素；(5)地球化学证据与地球动力学模拟都显示南海岩浆的形成受到周边俯冲带的影响。目前我国的海洋地球科学正在进入崭新的发展阶段，有望以南海为基点，开始拓展到周边大洋，通过主导大型研究计划以及建设我国大洋钻探平台，以提升我国在南海、西太平洋与印度洋海洋地质科学研究的实质性影响力与引领地位。  相似文献   

Growth and feeding habits of invasive Pseudorasbora parva in the Chabalang Wetland(Lhasa,China) and its trophic impacts on native fish     

DING Huiping  GU Xiaohong  ZHANG Zhiming  HUO Bin  LI Dapeng  XIE Congxin 《中国海洋湖沼学报》2019,(2)

In recent years, fish invasion has become one of the main reasons for the decline of nativefish stocks. Pseudorasbora parva is considered one of the major invasive species worldwide. The present study investigated the fish resources of the Chabalang Wetland(Lhasa, Tibet) during dif ferent seasons in2009 and 2013. Four hundred and twelve individuals were subsampled to estimate age, growth, and feeding habit of P. parva. Furthermore, food relationships between P. parva and the native Schizothoracinae fish were also examined. The results revealed a significant shift in species composition and community structure characterized by the disappearance of native fish and outbreak of non-native fish. The percentage of nonnative P. parva in the fish collections significantly increased from 33.64% in 2009 to 64.08% in 2013.The standard length(SL) ranged from 22.00 to 78.71 mm, and their age was 1–5 a. The von Bertalanf fy function was used to model the observed length-at-age data as Lt =112.19(1– e~(-0.1495(t +0.8012))) for females and as Lt =123.12(1– e~(-0.1500(t +0.7132))) for males. The results indicated that P. parva in Tibet has lower growth and mortality rates compared with that from the native ranges. Ninety-seven prey taxa belonging to 9 prey categories were identified in the gut of 38 P. parva. P. parva can be considered a generalized and opportunistic predator, competing with the native fish, especially Schizothorax o'connori, Schizopygopsis younghusbandi younghusbandi, and Ptychobarbus dipogon, for Bacillariophyta and Chironomid larvae.This is an important reason for the decline in native fish population.  相似文献   

Tidal Effects on the Bottom Thermal Front of North Yellow Sea Cold Water Mass near Zhangzi Island in Summer 2009     

XIA Yingying  BAO Xianwen  SONG Dehai  DING Yang  WAN Kai  YAN Yuhan 《中国海洋大学学报(英文版)》2019,(4)

Three seabed-mounted TD/CTD chains and two upward-looking acoustic Doppler current profilers(ADCPs) in the southwest of Zhangzi Island are used and a simultaneous cruise observation in the northern North Yellow Sea(NYS) is conducted to study temperature variation in the bottom thermal front zone of the NYS Cold Water Mass(NYSCWM) during the summer of 2009. In the flood-ebb tidal cycles, the bottom temperature decreases(increases) during flood(ebb) tides, which are dominated by the tidal-current induced horizontal advection. The ebb tide-induced temperature increase is larger than the flood tide-induced temperature decrease due to seasonal warming. In the spring-neap tidal cycles, the temperature and the vertical temperature structure show notable fortnightly variation from 16 July to 25 August. The bottom temperature increases from neap to spring tides and decreases from spring to neap. The Richardson number demonstrates strengthened vertical mixing during spring tides but enhanced stratification during neap tides. The spring-neap variation in vertical shear caused by tidal current is the dominant factor that induces the fortnightly variation in vertical mixing and thus bottom temperature.  相似文献   

多功能水合物沉积物三轴试验系统的研制与应用     

周家作  韦昌富  魏厚振  杨周洁  李力昕  李彦龙  丁根荣 《岩土力学》2020,41(1):342-352

研制了新型多功能水合物沉积物三轴试验系统，系统由供-排气模块、应力加载模块、温控模块、数据采集模块和辅助模块组成。利用泵驱动高压液体施加围压和轴压，加载方式有恒压、恒流、梯度增压、跟踪气压模式，可以实现不同条件下的水合物合成、三轴剪切。围压跟踪气压的功能使水合物生成过程中净围压不变，保证了试样初始条件一致性。根据围压液体积变化测量试样体积变形，通过设置回压泵压力变化方式控制水合物气压，可以模拟不同降压速率条件下的水合物沉积物分解变形。以泥质粉细砂和二氧化碳为材料制作了三轴试样，进行了水合物合成、三轴剪切和降压分解试验，检验了仪器的可靠性。  相似文献   

深圳市大气210Pb的活度浓度、沉降通量和沉积速率研究     

吴家堡  李涌  刘国卿  苏玲玲  丁敏霞 《地球化学》2021,50(2):219-225

2017年1月至12月,研究分析了深圳市大气气溶胶中210Pb的活度浓度、沉降通量和沉积速率.结果表明,深圳市大气210Pb比活度范围为0.59～4.72 mBq/m3(平均值为1.58 mBq/m3);沉降通量范围为0.45～1.31 Bq/(m2·d)(平均值为0.74 Bq/(m2·d));沉积速率范围为0.09～1.44 cm/s(平均值为0.71 cm/s).研究发现,深圳市大气210Pb比活度冬季高、夏季低,其季节变化受气象条件和不同气团来源综合影响.大气210Pb的沉降通量和沉积速率均与降水量呈明显的正相关关系,提示降雨是大气中210Pb去除的有效途径.  相似文献